Natural orifice surgical system, surgical method and computer device

Abstract

The embodiment of the present application is suitable for the field of computer-aided medical technology and data processing technology, and provides a natural orifice surgery system, a surgery method and a computer device. The system comprises: a preoperative planning unit configured to generate preoperative planning information, wherein the preoperative planning information comprises a planned surgery path; an intraoperative execution unit configured to construct a dynamic surgery environment model and a deformation prediction model, to perform real-time adjustment on the planned surgery path displayed in the dynamic surgery environment model based on intraoperative real-time data and deformation prediction data output by the deformation prediction model, and to perform natural orifice surgery operation according to the adjusted actual surgery path; and a postoperative evaluation unit configured to generate postoperative evaluation information of the natural orifice surgery according to operation data output by the intraoperative execution unit during the natural orifice surgery operation. The system can improve the accuracy of the NOTES surgery and reduce the difficulty of the surgery operation.

Description

Technical Field

[0001] This application belongs to the field of computer-aided medical technology and data processing technology, and in particular relates to a natural cavity surgery system, surgical method and computer equipment. Background Technology

[0002] Natural orifice transluminal endoscopic surgery (NOTES) allows for internal diagnosis and treatment through the body's natural cavities (such as the digestive and respiratory tracts), offering advantages such as no external incisions and rapid postoperative recovery. However, NOTES demands significantly higher precision than traditional minimally invasive surgery. For example, NOTES requires three-dimensional spatial positioning, tissue puncture, and the coordinated operation of complex instruments within tortuous and narrow cavities, placing extremely high demands on the surgeon's skill. Furthermore, current NOTES techniques rely heavily on the surgeon's experience and judgment, resulting in a relatively high error rate. Inexperienced surgeons require extensive training to achieve a stable level of proficiency, further hindering the widespread adoption of NOTES.

[0003] Therefore, how to improve the precision of NOTES surgery and use computer-aided technology to help doctors complete NOTES surgery efficiently and conveniently is a major problem that needs to be solved by those in the field. Summary of the Invention

[0004] In view of this, embodiments of this application provide a natural orifice surgery system, surgical method, and computer equipment to improve the precision of NOTES surgery and reduce the difficulty of surgical operation.

[0005] A first aspect of this application provides a natural orifice surgical system, comprising:

[0006] The preoperative planning unit is used to generate preoperative planning information based on three-dimensional image data of the lesion area and tissue elasticity data, the preoperative planning information including the planned surgical path;

[0007] The intraoperative execution unit is used to construct a dynamic surgical environment model and a deformation prediction model based on real-time intraoperative data. Based on the real-time intraoperative data and the deformation prediction data output by the deformation prediction model, it adjusts the planned surgical path displayed in the dynamic surgical environment model in real time and performs natural cavity surgical operations according to the adjusted actual surgical path. The real-time intraoperative data includes multimodal data, which is mapped to a unified coordinate system using a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the real-time intraoperative data.

[0008] The postoperative assessment unit is used to generate postoperative assessment information for this natural cavity surgery based on the operation data output by the intraoperative execution unit during the natural cavity surgery operation.

[0009] Optionally, the intraoperative execution unit is specifically used for:

[0010] Acquire historical surgical data and construct a system model based on the historical surgical data; map the multimodal data collected in real time during the operation to a unified time coordinate system; construct a dynamic surgical environment model based on the system model and the multimodal data in the time coordinate system;

[0011] Feature values ​​are extracted from the multimodal data and input into a spatiotemporal graph convolutional network to obtain information on the deformation patterns of internal cavity tissues. A deformation prediction model based on an LSTM-Transformer neural network is constructed based on the deformation patterns of internal cavity tissues. The graph structure in the spatiotemporal graph convolutional network is composed of various internal cavities.

[0012] Optionally, the multimodal data includes endoscopic video stream data and tissue biomechanical signal data acquired during the natural orifice surgery. The endoscopic video stream data includes multiple images of the internal cavity, and the tissue biomechanical signal data includes contact forces and deformations of the internal cavity tissues. The intraoperative execution unit is further specifically used for:

[0013] Contact forces and deformations of internal cavity tissues are collected using pressure sensors installed at the end of surgical instruments.

[0014] Determine the internal cavity image corresponding to the deformation;

[0015] The contact force, deformation, and feature values ​​of the internal cavity tissue at the same time are extracted, and the feature values ​​of the contact force, deformation, and internal cavity image at the same time are correlated and used as input data for the spatiotemporal graph convolutional network.

[0016] Optionally, the intraoperative execution unit is further specifically used for:

[0017] Monitor the tissue biomechanical signal data in the intraoperative real-time data;

[0018] The tissue mechanical signal data is input into the deformation prediction model to obtain the displacement field and strain field of the internal cavity; wherein, the displacement field is used to characterize the positional change of the internal cavity after deformation, and the strain field is used to characterize the degree of deformation at each position point in the internal cavity;

[0019] The deformation of the inner cavity is predicted based on the displacement field and the strain field, thus obtaining deformation prediction data for the inner cavity.

[0020] Optionally, the intraoperative execution unit is further specifically used for:

[0021] Monitor the endoscopic video stream data in the intraoperative real-time data;

[0022] The risk warning model is used to compare each frame of the endoscopic video stream data to obtain the comparison results; wherein, the risk warning model is trained based on sample images marked with bleeding and perforation features;

[0023] Based on the comparison results, real-time warnings are issued for operations with a risk of bleeding and / or perforation.

[0024] Optionally, the intraoperative execution unit is further specifically used for:

[0025] The intraoperative real-time data is processed using a deep reinforcement learning model to output the operational parameters during the natural cavity surgery process; wherein, the deep reinforcement learning model is based on a transfer learning framework and is trained on historical surgical data.

[0026] Optionally, the operating parameters include parameters characterizing the operating trajectory of the actual surgical path, and the intraoperative execution unit is further specifically used for:

[0027] Monitor the mechanical feedback data of the surgical instrument tip in the real-time intraoperative data;

[0028] The operation trajectory is corrected based on the mechanical feedback data.

[0029] Optionally, the preoperative planning unit is specifically used for:

[0030] Acquire three-dimensional image data of the lesion area obtained by a three-dimensional imaging device and tissue elasticity data of the lesion area obtained by an elasticity imaging device;

[0031] Based on the three-dimensional image data and the tissue elasticity data, a three-dimensional model of the lesion area is constructed;

[0032] Surgical planning is performed in a three-dimensional model of the lesion area to obtain preoperative planning information.

[0033] Optionally, the postoperative assessment unit is specifically used for:

[0034] A quality assessment model was used to evaluate the operation process of this natural orifice surgery to obtain postoperative evaluation information. The quality assessment model was generated based on the operation data output by the intraoperative execution unit during previous natural orifice surgery operations. The operation data output during this natural orifice surgery operation will be used to update the quality assessment model.

[0035] A second aspect of this application provides a natural cavity surgery method, including:

[0036] A dynamic surgical environment model and a deformation prediction model are constructed based on real-time intraoperative data.

[0037] Based on the intraoperative real-time data and the deformation prediction data output by the deformation prediction model, the planned surgical path displayed in the dynamic surgical environment model is adjusted in real time.

[0038] Perform natural cavity surgery procedures according to the adjusted actual surgical path;

[0039] The intraoperative real-time data includes multimodal data, which is mapped to a unified coordinate system through a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the intraoperative real-time data.

[0040] A third aspect of this application provides a computer device for implementing operations performed by various units of a natural cavity surgical system as described in any of the first aspects; the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device performs the method described in the second aspect above.

[0041] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the method performed by each unit of the natural cavity surgical system as described in any of the first aspects above.

[0042] A fifth aspect of this application provides a computer program product, including a computer program that, when run, causes the operations performed by each unit of the natural cavity surgical system described in any of the first aspects to be executed.

[0043] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0044] This application embodiment, by fusing multimodal data, can construct an environmental perception model characterizing the actual surgical environment. Combining the deformation prediction model's prediction results of tissue displacement or deformation, the pre-planned surgical path can be adjusted in real time, ensuring that surgical instruments move along a safe path. Furthermore, by comparing the current surgical operation with the pre-operative plan in real time, subsequent operations can be dynamically corrected, further improving surgical precision and avoiding error accumulation. Applying the natural orifice surgery system provided in this application embodiment can significantly improve surgical precision and convenience, while reducing the difficulty of surgical procedures. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a natural orifice surgical system provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of a natural cavity surgical method provided in an embodiment of this application;

[0048] Figure 3 This is an organizational diagram of a natural cavity surgical system provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of a natural cavity surgical procedure provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0052] As mentioned earlier, NOTES surgery in related technologies relies heavily on the surgeon's experience and judgment, resulting in a high error rate. In developing the solution presented in this application, the applicant discovered that current NOTES surgery does not fully leverage the potential value of surgical data, leading to isolated data from various modules. For example, preoperative 3D imaging data, intraoperative imaging data, and intraoperative instrument mechanical signal data are all stored independently, and related technologies lack a unified solution for processing various surgical data from NOTES surgery.

[0053] To address this issue, this application provides a natural orifice surgical system that can fully leverage the potential value of various surgical data from NOTES surgery to guide NOTES procedures, improve surgical precision, and reduce the difficulty of surgical operations.

[0054] Specifically, the natural orifice surgery system provided in this application embodiment can integrate multimodal data (such as geometric information from endoscopic images, dynamic feedback of instrument mechanics (contact force / torque), and blood flow status of physiological parameters) in real time during surgery to construct an environmental perception model; it uses deformation compensation algorithms to predict tissue displacement trends (such as intestinal peristalsis) and reduce positioning errors during intraoperative operations. Secondly, the system can analyze historical surgical data through transfer learning, extract high-risk operation patterns (such as cutting angles in areas adjacent to blood vessels), generate corresponding avoidance strategies, and ensure the safety of surgical operations; it uses reinforcement learning models to optimize surgical paths based on real-time intraoperative data, shortening the time required for complex surgical operations (such as intestinal suturing). In addition, the natural orifice surgery system provided in this application embodiment can also compare the differences between the surgical operation results and preoperative planning (such as deviations in determining the resection range) in real time based on data closure, dynamically correct subsequent operation parameters, and avoid error accumulation; the postoperative analysis function provided by the system can quantify the quality of this surgical operation (such as determining the tension distribution of sutures), form a physician competence assessment model, and assist in improving the surgeon's surgical operation level and accelerating the inheritance of NOTES surgical experience.

[0055] The technical solution of this application will be described below through specific embodiments.

[0056] Reference Figure 1 This illustration shows a schematic diagram of a natural orifice surgical system provided in an embodiment of this application. Specifically, Figure 1 The natural orifice surgical system 100 shown may include a preoperative planning unit 101, an intraoperative execution unit 102, and a postoperative evaluation unit 103. Wherein:

[0057] The preoperative planning unit 101 is used to generate preoperative planning information based on three-dimensional imaging data and tissue elasticity data of the lesion area. The preoperative planning information includes the planned surgical path.

[0058] The intraoperative execution unit 102 is used to construct a dynamic surgical environment model and a deformation prediction model based on real-time intraoperative data. Based on the real-time intraoperative data and the deformation prediction data output by the deformation prediction model, it adjusts the planned surgical path displayed in the dynamic surgical environment model in real time and performs natural cavity surgical operations according to the adjusted actual surgical path. The real-time intraoperative data includes multimodal data, which is mapped to a unified coordinate system using a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the real-time intraoperative data.

[0059] The postoperative assessment unit 103 is used to generate postoperative assessment information for this natural cavity surgery based on the operation data output by the intraoperative execution unit during the natural cavity surgery operation.

[0060] The following is a detailed description of the specific operations performed by each unit of the natural cavity surgical system 100 and the process of completing surgery using the natural cavity surgical system 100.

[0061] In the embodiments of this application, Figure 1 The natural orifice surgery system 100 shown can be used to perform natural orifice surgeries. Unless otherwise specified, natural orifice surgery mentioned in the embodiments of this application can refer to natural orifice endoscopic surgery, i.e., NOTES surgery.

[0062] The preoperative planning unit 101 in the natural orifice surgery system 100 is mainly used for preoperative surgical planning. The preoperative planning unit 101 can obtain relevant data about the lesion area and, in conjunction with appropriate surgical strategies, plan the surgical procedure. This relevant data may include three-dimensional imaging data, tissue elasticity data, etc.

[0063] For example, three-dimensional imaging data of the patient's lesion area can be acquired using three-dimensional imaging equipment, such as CT or MRI equipment, and tissue elasticity data of the same lesion area can be acquired using elastography equipment. Specifically, internal cavity data can be non-invasively obtained using the shear wave velocity method, and then elastography can be performed using ultrasound or MRI equipment to obtain tissue elasticity data.

[0064] The preoperative planning unit 101 can perform three-dimensional modeling based on the aforementioned three-dimensional image data and tissue elasticity data to construct a three-dimensional model of the lesion area. In this way, the surgeon can combine surgical strategies with the three-dimensional model of the lesion area to plan the surgery and obtain preoperative planning information, which is the surgical plan to be followed during the subsequent actual surgery. The surgeon can perform the surgery according to the planned surgical plan with the assistance of the intraoperative execution unit 102. In NOTES surgery, instruments (such as endoscopes) need to be used to enter the patient's body through natural cavities and reach the lesion area for surgery. Therefore, the preoperative planning information for NOTES surgery includes a planned surgical path, which may include the planned path for the instruments to enter the patient's body through natural cavities, or the trajectory or path of the instruments during the actual surgery. In addition to the surgical path, the preoperative planning information may also include parameters related to other operations during the surgical execution process, such as tissue resection location and suturing method. This application embodiment does not limit this.

[0065] In this embodiment, when assisting the surgeon in performing NOTES surgery, the intraoperative execution unit 102 can first construct a dynamic surgical environment model based on real-time intraoperative data. This dynamic surgical environment model can be an environmental perception model constructed based on environmental perception. Through this dynamic surgical environment model, the actual environment of the surgical area, such as the lesion, can be simulated in real time, providing decision-making reference for the surgeon's operation.

[0066] In one possible implementation of the embodiments of this application, Figure 1 The natural orifice surgery system 100 shown may include a display unit, through which a dynamic surgical environment model constructed based on real-time intraoperative data can be displayed, allowing doctors to intuitively understand the relevant conditions of the surgical area.

[0067] In this embodiment, the intraoperative real-time data may include multimodal data, such as endoscopic video stream data, instrument trajectory data, tissue biomechanical signal data, and blood pressure and pulse signal data. Specifically, the endoscopic video stream data may be multiple images of the internal cavity acquired by an endoscope or a camera on the endoscope; the instrument trajectory data may be the trajectory of the surgical instrument's distal end as it enters the internal cavity and moves during the surgical procedure; the tissue biomechanical signal data may include data such as contact force and deformation of various tissues within the internal cavity, which can be acquired by a pressure sensor installed at the distal end of the surgical instrument; and the blood pressure and pulse signal data may include the patient's blood pressure and pulse during the surgical procedure.

[0068] The intraoperative execution unit 102 can construct a dynamic surgical environment model by mapping multimodal data to a unified coordinate system.

[0069] Specifically, a system model can be built based on historical surgical data. Then, the multimodal data collected in real time during the operation can be mapped to a unified time coordinate system to ensure that the multimodal data is aligned on the time axis. Thus, a dynamic surgical environment model that can reflect the real situation of the surgical environment in real time can be constructed based on the already built system model and the multimodal data in the time coordinate system.

[0070] In this embodiment, the system model can be pre-constructed before the operation. For example, a system model can be pre-constructed by acquiring relevant surgical data from previous NOTES surgeries. In this way, during the actual surgical procedure, by combining the processing of real-time intraoperative data, and based on the pre-constructed system model, real-time surgical data such as the collected endoscopic video stream data and instrument movement trajectory data can be fused into the system model through environmental perception, thereby constructing a dynamic surgical environment model.

[0071] In one possible implementation of this application, a spatiotemporal alignment algorithm can be used to process the multimodal data and map it to a unified time coordinate system. For example, Kalman filtering or similar processing methods can be used to process the multimodal data and map it to a unified coordinate system.

[0072] After a dynamic surgical environment model is constructed and displayed through the display unit of the natural cavity surgical system 100, the planned surgical path in the preoperative planning information can also be displayed in the model, thereby guiding the surgeon to move the instruments along the planned surgical path during the surgical procedure.

[0073] Typically, during NOTES surgical procedures, the movement of instruments can affect adjacent tissues; for example, instrument movement may cause deformation of internal cavity tissues. Tissue deformation caused by sharp instruments may also lead to bleeding and perforation accidents. Therefore, in this embodiment, the intraoperative execution unit 102 can also construct a deformation prediction model to predict the deformation of internal cavity tissues in real time.

[0074] In the embodiments of this application, the deformation prediction model can also be constructed based on real-time intraoperative multimodal data.

[0075] In one possible implementation of this application embodiment, the deformation prediction model can be constructed by combining a spatiotemporal graph convolutional network and an LSTM-Transformer neural network. The graph structure in the spatiotemporal graph convolutional network can be composed of various internal cavities.

[0076] Specifically, feature values ​​extracted from multimodal data can be used as input data for a spatiotemporal graph convolutional network (SPCRN). Each feature value is then input into the SPCRN. Through processing by the SPCRN, the deformation patterns of internal cavity tissues are obtained. Based on this, and combined with an LSTM-Transformer neural network, a deformation prediction model based on the obtained internal cavity tissue deformation patterns is constructed.

[0077] The multimodal data required to construct the deformation prediction model can include endoscopic video stream data and tissue biomechanical signal data acquired during this natural cavity surgery procedure. The endoscopic video stream data includes multiple images of the internal cavity, while the tissue biomechanical signal data can include contact forces and deformations of the internal cavity tissues.

[0078] In one possible implementation of this application, the contact force and deformation of the internal cavity tissue can be collected by a pressure sensor installed at the end of a surgical instrument. Abnormal data must be discarded. During data processing, various types of data can be aligned on a time axis. For example, for a specific deformation, its corresponding internal cavity image is determined. Then, feature values ​​of the contact force, deformation, and internal cavity image of the internal cavity tissue at the same time are extracted and correlated as input data for a spatiotemporal graph convolutional network. By combining the contact force and deformation of the internal cavity tissue with the internal cavity image captured by an endoscopic camera, tissue deformation characteristics can be tracked, thereby outputting the tissue deformation pattern through the spatiotemporal graph convolutional network.

[0079] The tissue deformation patterns output by the spatiotemporal graph convolutional network can be processed using an LSTM-Transformer neural network to construct a deformation prediction model. This LSTM-Transformer neural network can be pre-built and processed in conjunction with multimodal data from the actual surgery to construct the deformation prediction model required for the current operation, thereby reducing the time required for model construction.

[0080] In this embodiment, the intraoperative execution unit 102 can monitor the tissue mechanics signal data in real-time intraoperative data and input the tissue mechanics signal data into the aforementioned deformation prediction model to obtain the displacement field and strain field of the internal cavity. The displacement field can be used to characterize the positional change of the internal cavity after deformation, and the strain field can be used to characterize the degree of deformation at various points within the internal cavity. Thus, the deformation prediction model can predict the deformation of the internal cavity based on the displacement field and strain field, obtaining deformation prediction data for the internal cavity. This deformation prediction data can represent the possible deformation of the internal cavity tissue caused by various factors during the surgical procedure. For example, the impact of surgical instruments on surrounding tissues during movement can lead to tissue deformation. The deformation prediction model can output specific deformation values ​​and other data that may result from the deformation.

[0081] In another possible implementation of this application embodiment, the intraoperative execution unit 102 may also provide a risk warning model to provide warnings of possible risks such as bleeding and perforation.

[0082] In this embodiment, the risk warning model can be trained based on sample images labeled with bleeding and perforation features. Specifically, some images of internal cavities can be pre-acquired, and images and locations showing bleeding and puncture can be labeled. The labeled images can be used as sample images for training the risk warning model. The risk warning model is obtained by training a convolutional neural network (CNN) and configured in the natural orifice surgery system 100.

[0083] In this way, during the actual surgical procedure, the intraoperative execution unit 102 can monitor the endoscopic video stream data in real time, and use a risk warning model to compare each frame of the endoscopic video stream data to obtain the comparison results. Based on the comparison results, real-time warnings are issued for procedures with bleeding and / or perforation risks. Specifically, the risk warning model can locate the possible bleeding site based on the spatial features of each frame of the image, and determine the possible perforation risk by detecting the deformation of the internal cavity in each frame of the image.

[0084] In this embodiment of the application, the doctor can Figure 1 With the assistance of the natural cavity surgery system 100 shown, surgical procedures are performed in conjunction with the aforementioned dynamic surgical environment model. Furthermore, during the surgical procedure, tissue deformation can be predicted based on prompts from the deformation prediction model, thereby adjusting the pre-planned surgical path and performing the surgery according to the adjusted path. Simultaneously, the risk warning model can provide real-time warnings regarding potential bleeding and perforation risks during the surgical procedure, ensuring the safety of the procedure.

[0085] In one possible implementation of this application embodiment, the natural cavity surgery system 100 can also train a deep reinforcement learning model (DQN) based on the processing of historical surgical data, thereby guiding the surgeon's specific operational behaviors during the surgery. For example, how to insert the needle, how to cut tissue, etc.

[0086] In this embodiment, the deep reinforcement learning model can be trained on historical surgical data using a transfer learning framework. For example, after each NOTES surgery, the corresponding surgical data can be stored uniformly. The surgical system can combine a large number of NOTES surgery cases (e.g., more than 10,000 cases) to train a deep reinforcement learning model that can guide doctors in actual surgical procedures.

[0087] The intraoperative execution unit 102 can use a deep reinforcement learning model to process real-time intraoperative data and output relevant operational parameters during the natural cavity surgery, such as instrument insertion angle and tissue cutting data. Surgeons can operate according to the parameters output by the deep reinforcement learning model, improving surgical precision and reducing the difficulty of the procedure.

[0088] In this embodiment of the application, the relevant data during the natural cavity surgery can also be used to retrain the deep reinforcement learning model after the surgery is completed, so as to further improve the accuracy and reliability of the deep reinforcement learning model and the intraoperative execution unit 102.

[0089] During the surgical procedure performed by the surgeon according to the operational parameters output by the deep reinforcement learning model, the intraoperative execution unit 102 can also dynamically adjust the surgical path (such as avoiding areas with dense blood vessels) by combining real-time intraoperative data, thereby reducing path deviation. Specifically, the intraoperative execution unit 102 can compare the deformation prediction data output by the deformation prediction model with the pre-set deformation threshold, and adjust the preoperative planned path according to the comparison results to avoid the risk of puncture.

[0090] In one possible implementation of this application embodiment, the operational parameters output by the deep reinforcement learning model may include parameters representing the operational trajectory of the actual surgical path. The intraoperative execution unit 102 can monitor real-time intraoperative data to correct the operational trajectory, further improving the accuracy and safety of the surgical operation.

[0091] Specifically, real-time intraoperative data may include mechanical feedback data from the distal end of the surgical instrument, which can be obtained through force-position hybrid control (FPC) at the instrument distal end. The intraoperative execution unit 102 can dynamically correct the instrument's operating trajectory based on the mechanical feedback data to ensure that the instrument's movement trajectory meets safety requirements.

[0092] The Natural Orifice Surgical System 100 provided in this application embodiment can guide doctors to conveniently complete NOTES surgery through the relevant processing of real-time intraoperative data by the intraoperative execution unit 102, and can ensure the accuracy and safety of NOTES surgery.

[0093] In another possible implementation of the embodiments of this application, Figure 1 The Natural Orifice Surgical System 100 can also be equipped with a quality assessment model, which can be used to evaluate the operation process of this natural orifice surgery, obtain postoperative assessment information, evaluate the surgeon's surgical skill and surgical effect, and reuse relevant best practices to provide a reference for other surgeons performing NOTES surgery.

[0094] In this embodiment, the quality assessment model can be generated based on the operational data output by the intraoperative execution unit during each natural orifice surgery procedure; that is, historical surgical data can be used to construct the quality assessment model. For example, data such as suture tension distribution and suture position during each surgical procedure can be used to update the quality assessment model, including the operational data output during the current natural orifice surgery procedure.

[0095] Based on the aforementioned natural orifice surgical system 100, this application also provides a natural orifice surgical method. (Refer to...) Figure 2 The diagram illustrates a natural orifice surgical method provided in an embodiment of this application. This method may specifically include the following steps:

[0096] S201. Construct a dynamic surgical environment model and a deformation prediction model based on real-time intraoperative data.

[0097] S202. Based on the intraoperative real-time data and the deformation prediction data output by the deformation prediction model, the planned surgical path displayed in the dynamic surgical environment model is adjusted in real time.

[0098] S203. Perform natural cavity surgery according to the adjusted actual surgical path.

[0099] The entity executing this method can be Figure 1 The natural orifice surgery system 100 shown above can assist doctors in completing various NOTES surgeries by executing the various steps of this method. The natural orifice surgery system 100 is configured in a computer device, which can be a desktop computer, a cloud server, or other similar devices; this application embodiment does not limit this to any particular type.

[0100] It should be understood that Figure 2The surgical method shown is the process of performing surgical operations using the aforementioned natural cavity surgical system 100, that is... Figure 1 The relevant operations are performed by the intraoperative execution unit 102. In practical applications of this method, the preoperative planning unit 101 can also be used to plan the surgical procedure. After the surgery is completed, the postoperative evaluation unit 103 can also be used to evaluate the surgical process.

[0101] Since the execution process of this method is similar to the process of the execution unit 102 performing related operations in the aforementioned system embodiment, the relevant introduction of this method can be found in the corresponding part of the aforementioned system embodiment, and will not be repeated here.

[0102] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0103] To facilitate understanding, the process of performing natural cavity surgery using the natural cavity surgery system provided in the embodiments of this application will be described below with reference to a complete example.

[0104] like Figure 3 The diagram shown is an organizational structure diagram of a natural cavity surgical system provided in an embodiment of this application. Figure 3 The system shown includes a data acquisition device 301, a data processing device 302, and a surgical operation device 303. The data acquisition device 301 may include a 3D imaging device 3011, an elastography device 3012, and a blood pressure and pulse monitor 3013. The 3D imaging device 3011, such as a CT scanner or MRI scanner, is used to scan the patient's lesion area to obtain 3D image data, including the anatomical structure of the lesion area. The elastography device 3012 can acquire tissue elasticity data of the lesion area through elastography. The blood pressure and pulse monitor 3013 can monitor the patient's blood pressure, pulse, and other data in real time.

[0105] Data processing device 302 can be an electronic device or computer device with data processing capabilities. For example... Figure 3 As shown, the data processing device 302 may include multiple programs, each of which is used to implement one or more data processing functions.

[0106] like Figure 3As shown, the data processing device 302 includes a data collection program 3021, a 3D modeling program 3022, a surgical planning program 3023, an intraoperative guidance program 3024, and an early warning program 3025. The data collection program 3021 can collect various data during the surgical process, including real-time intraoperative data. For example, the data collection program 3021 can record intraoperative videos, collect instrument movement trajectory data, tissue biomechanical signal data, and blood pressure and pulse signal data. The 3D modeling program 3022 can be used to create 3D models. For example, the 3D modeling program 3022 can achieve… Figure 1 The preoperative planning unit 101 performs some functions, constructing a three-dimensional model of the lesion area through processing three-dimensional image data and tissue elasticity data. The surgical planning program 3023 can also achieve this. Figure 1 The preoperative planning unit 101 utilizes a reinforcement learning-based decision-making system to plan the optimal surgical path. Building upon this, the surgical planning program 3023 can also perform some functions of the intraoperative execution unit 102. By comparing the planned surgical path with the actual surgical path, it corrects the path planning during the operation and optimizes the instrument's operating path or trajectory using intraoperative avoidance strategies. The intraoperative guidance program 3024 can output tissue deformation prediction data based on the aforementioned deformation prediction model and real-time mechanical signals, such as predicting the deformation of soft tissues like the gastrointestinal tract, to adjust the surgical path. The early warning program 3025 uses a convolutional neural network to analyze each frame of the endoscopic video stream data, providing real-time warnings of potential bleeding and puncture risks to ensure surgical safety.

[0107] Surgical operating equipment 303 can be the equipment used to actually perform NOTES surgical procedures. The relevant functions of this equipment can be realized through NOTES program 3031. This includes performing the surgery, collecting motion data of surgical instruments and endoscopic video stream data during the surgery, and processing various types of data to complete the entire NOTES surgery.

[0108] based on Figure 3 The natural cavity surgical system shown in the diagram allows for the completion of a full NOTES procedure as follows: Figure 4 As shown. (Refer to...) Figure 4 This diagram illustrates a natural cavity surgical procedure according to an embodiment of this application. Figure 4 As shown, the entire NOTES procedure includes preoperative planning, intraoperative execution, and postoperative evaluation.

[0109] During the preoperative planning phase, the surgical system can acquire imaging data such as the anatomical structure of the lesion area using CT or MRI equipment, and obtain the tissue elasticity of the lesion area through elastography. Based on this, the surgical system can create a three-dimensional (3D) model of the lesion area and, in conjunction with the surgical strategy, complete the preoperative planning of the surgical plan.

[0110] During the intraoperative execution phase, the surgical system can collect relevant data in real time, such as endoscopic video stream data, instrument movement trajectory data, tissue biomechanical signal data, and blood pressure and pulse signal data. Different data can be processed in different ways.

[0111] like Figure 4 As shown, endoscopic video stream data, instrument trajectory data, and tissue biomechanical signal data can be used to construct dynamic surgical environment models and deformation prediction models. Through intraoperative feedback and correction, corresponding guidance is provided for surgical operations. For example, based on the above data, deformation prediction models can be combined to predict organ and tissue deformation and compensate for the data in real time. The surgical system provided in this application embodiment can achieve sub-millimeter-level organ deformation compensation. Furthermore, based on the above data, deep learning reinforcement models can be combined to correct the next instrument path, ensuring the accuracy and safety of the surgical instrument trajectory. In addition, human-machine collaboration can be combined to achieve coupled tactile and visual feedback, providing guidance for the surgeon's operation.

[0112] like Figure 4 As shown, intraoperative early warning can be based on endoscopic video stream data and blood pressure and pulse signal data. For example, a risk warning model can be used to provide real-time warnings of potential bleeding and perforation risks.

[0113] During the postoperative evaluation phase, the surgical system can store all operational data from the surgery and analyze and compare the pre-planned surgical path with the actual surgical path to optimize the path planning strategy. Furthermore, embodiments of this application can utilize a surgical data database to accumulate data from previous surgeries, enabling the creation of a surgical simulation learning model. This model allows doctors to simulate surgical procedures, helping them accumulate surgical experience and rapidly improve their surgical skills.

[0114] The natural orifice surgery system provided in this application can achieve a systematic improvement in the accuracy of NOTES surgery through full-process optimization of surgical data acquisition, dynamic modeling, intelligent decision-making and closed-loop feedback.

[0115] Reference Figure 5 The diagram illustrates a computer device provided in an embodiment of this application. Figure 5As shown, the computer device 500 in this embodiment includes: a processor 510, a memory 520, and a computer program 521 stored in the memory 520 and executable on the processor 510. When the processor 510 executes the computer program 521, it implements the steps in the various embodiments of the above-described natural cavity surgery method, for example... Figure 2 The steps S201 to S203 are shown. Alternatively, when the processor 510 executes the computer program 521, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 1 The functions of units 101 to 103 are shown.

[0116] For example, the computer program 521 can be divided into one or more modules / units, which are stored in the memory 520 and executed by the processor 510 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 521 in the computer device 500. For example, the computer program 521 can be divided into a preoperative planning unit, an intraoperative execution unit, and a postoperative evaluation unit, with the specific functions of each module as follows:

[0117] The preoperative planning unit is used to generate preoperative planning information based on three-dimensional image data of the lesion area and tissue elasticity data, the preoperative planning information including the planned surgical path;

[0118] The intraoperative execution unit is used to construct a dynamic surgical environment model and a deformation prediction model based on real-time intraoperative data. Based on the real-time intraoperative data and the deformation prediction data output by the deformation prediction model, it adjusts the planned surgical path displayed in the dynamic surgical environment model in real time and performs natural cavity surgical operations according to the adjusted actual surgical path. The real-time intraoperative data includes multimodal data, which is mapped to a unified coordinate system using a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the real-time intraoperative data.

[0119] The postoperative assessment unit is used to generate postoperative assessment information for this natural cavity surgery based on the operation data output by the intraoperative execution unit during the natural cavity surgery operation.

[0120] The computer device 500 may be a device capable of realizing the relevant functions of each unit in the aforementioned natural cavity surgical system embodiments. The computer device 500 may be a desktop computer, a cloud server, or other similar devices. The computer device 500 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art will understand that...Figure 5 This is merely one example of computer device 500 and does not constitute a limitation on computer device 500. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device 500 may also include input / output devices, network access devices, buses, etc.

[0121] The processor 510 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0122] The memory 520 can be an internal storage unit of the computer device 500, such as a hard disk or memory of the computer device 500. The memory 520 can also be an external storage device of the computer device 500, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 500. Furthermore, the memory 520 can include both internal and external storage units of the computer device 500. The memory 520 is used to store the computer program 521 and other programs and data required by the computer device 500. The memory 520 can also be used to temporarily store data that has been output or will be output.

[0123] This application also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the methods described in the foregoing embodiments or implements the operations performed by the various units in the foregoing system embodiments.

[0124] This application also discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, it implements the methods described in the foregoing embodiments or implements the operations performed by the various units in the foregoing system embodiments.

[0125] This application also discloses a computer program product, including a computer program that, when run on a computer, causes the computer to execute the methods described in the foregoing embodiments or to implement the operations performed by the units in the foregoing system embodiments.

[0126] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A natural cavity surgical system, characterized in that, include: The preoperative planning unit is used to generate preoperative planning information based on three-dimensional image data of the lesion area and tissue elasticity data, the preoperative planning information including the planned surgical path; The intraoperative execution unit is used to construct a dynamic surgical environment model and a deformation prediction model based on real-time intraoperative data. Based on the real-time intraoperative data and the deformation prediction data output by the deformation prediction model, it adjusts the planned surgical path displayed in the dynamic surgical environment model in real time and performs natural cavity surgical operations according to the adjusted actual surgical path. The real-time intraoperative data includes multimodal data, which is mapped to a unified coordinate system using a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the real-time intraoperative data. The postoperative assessment unit is used to generate postoperative assessment information for this natural cavity surgery based on the operation data output by the intraoperative execution unit during the natural cavity surgery operation. Specifically, the intraoperative execution unit is used for: Acquire historical surgical data and construct a system model based on the historical surgical data; map the multimodal data collected in real time during the operation to a unified time coordinate system; construct a dynamic surgical environment model based on the system model and the multimodal data in the time coordinate system; Feature values ​​are extracted from the multimodal data and input into a spatiotemporal graph convolutional network to obtain information on the deformation patterns of internal cavity tissues. A deformation prediction model based on an LSTM-Transformer neural network is constructed based on the deformation patterns of internal cavity tissues. The graph structure in the spatiotemporal graph convolutional network is composed of various internal cavities.

2. The surgical system according to claim 1, characterized in that, The multimodal data includes endoscopic video stream data and tissue biomechanical signal data acquired during this natural cavity surgery. The endoscopic video stream data includes multiple images of the internal cavity, and the tissue biomechanical signal data includes contact forces and deformations of the internal cavity tissues. The intraoperative execution unit is further specifically used for: Contact forces and deformations of internal cavity tissues are collected using pressure sensors installed at the end of surgical instruments. Determine the internal cavity image corresponding to the deformation; The contact force, deformation, and feature values ​​of the internal cavity tissue at the same time are extracted, and the feature values ​​of the contact force, deformation, and internal cavity image at the same time are correlated and used as input data for the spatiotemporal graph convolutional network.

3. The surgical system according to claim 1 or 2, characterized in that, The intraoperative execution unit is also specifically used for: Monitor the tissue biomechanical signal data in the intraoperative real-time data; The tissue mechanical signal data is input into the deformation prediction model to obtain the displacement field and strain field of the internal cavity; wherein, the displacement field is used to characterize the positional change of the internal cavity after deformation, and the strain field is used to characterize the degree of deformation at each position point in the internal cavity; The deformation of the inner cavity is predicted based on the displacement field and the strain field, thus obtaining deformation prediction data for the inner cavity.

4. The surgical system according to claim 3, characterized in that, The intraoperative execution unit is also specifically used for: Monitor the endoscopic video stream data in the intraoperative real-time data; The risk warning model is used to compare each frame of the endoscopic video stream data to obtain the comparison results; wherein, the risk warning model is trained based on sample images marked with bleeding and perforation features; Based on the comparison results, real-time warnings are issued for operations with a risk of bleeding and / or perforation.

5. The surgical system according to any one of claims 1, 2, or 4, characterized in that, The intraoperative execution unit is also specifically used for: The intraoperative real-time data is processed using a deep reinforcement learning model to output the operational parameters during the natural cavity surgery process; wherein, the deep reinforcement learning model is based on a transfer learning framework and is trained on historical surgical data.

6. The surgical system according to claim 5, characterized in that, The operating parameters include parameters characterizing the operating trajectory of the actual surgical path, and the intraoperative execution unit is further specifically used for: Monitor the mechanical feedback data of the surgical instrument tip in the real-time intraoperative data; The operation trajectory is corrected based on the mechanical feedback data.

7. The surgical system according to any one of claims 1, 2, 4, or 6, characterized in that, The preoperative planning unit is specifically used for: Acquire three-dimensional image data of the lesion area obtained by a three-dimensional imaging device and tissue elasticity data of the lesion area obtained by an elasticity imaging device; Based on the three-dimensional image data and the tissue elasticity data, a three-dimensional model of the lesion area is constructed; Surgical planning is performed in a three-dimensional model of the lesion area to obtain preoperative planning information.

8. The surgical system according to claim 7, characterized in that, The postoperative assessment unit is specifically used for: A quality assessment model was used to evaluate the operation process of this natural orifice surgery to obtain postoperative evaluation information. The quality assessment model was generated based on the operation data output by the intraoperative execution unit during previous natural orifice surgery operations. The operation data output during this natural orifice surgery operation will be used to update the quality assessment model.

9. A computer device for implementing operations performed by various units of a natural cavity surgical system as described in any one of claims 1 to 8; said computer device comprising a memory, a processor, and a computer program stored in said memory and executable on said processor, characterized in that, When the processor executes the computer program, the computer device performs the following method: A dynamic surgical environment model and a deformation prediction model are constructed based on real-time intraoperative data. Based on the intraoperative real-time data and the deformation prediction data output by the deformation prediction model, the planned surgical path displayed in the dynamic surgical environment model is adjusted in real time. Perform natural cavity surgery procedures according to the adjusted actual surgical path; The intraoperative real-time data includes multimodal data, which is mapped to a unified coordinate system through a spatiotemporal alignment algorithm to construct the dynamic surgical environment model. The deformation prediction model is used to predict tissue deformation based on the intraoperative real-time data.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is run, the operations performed by each unit of the natural cavity surgical system as described in any one of claims 1 to 8 are executed.

Images